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Executive  Summary 


The  Integrated  Risk  Assessment  and  Management  Model  (IRAMM)  is  used  as  part  of  a 
decision-support  process  that  the  Department  of  Defense  (DOD)  and,  more  broadly,  the  United 
States  Government  (USG)  now  employ  to  assess  risks  and  identify  cost-effective  risk  mitigation 
strategies  in  the  area  of  non-fuel  strategic  materials.  The  strategic  material  decision-support 
process  (the  Risk  Assessment  and  Mitigation  Framework  for  Strategic  Materials  (RAMF-SM))  is 
used  to  assess  risks  and  set  priorities  for  preparedness  programs  designed  to  mitigate  potential 
strategic  material  shortfalls.  IRAMM  provides  the  strategic  risk  context  for  RAMF-SM  risk 
assessments. 

IRAMM  is  a  flexible  approach  to  strategic  risk  assessment  that  employs  structured, 
interactive  one-on-one  interviews  with  national  security  experts.  The  interviews  are  structured 
around  four  categories  known  as  Challenge  Areas  that  are  defined  so  that  together  they  cover  a 
full  range  of  potential  operations  conducted  by  the  U.S.  military  during  the  next  decade.  The 
results  from  IRAMM  were  used  as  inputs  to  the  RAMF-SM  process  that  helped  produce  the 
Strategic  and  Critical  Materials  2013  Report  on  Stockpile  Requirements  to  Congress.  IRAMM 
interviews  provided  assessments  of  the  probabilities  of  potential  scenarios  that  pose  strategic  risk 
to  the  Nation.  These  scenarios  included  but  were  not  limited  to  the  scenarios  specified  by 
Congress  for  the  2013  Report  on  Stockpile  Requirements.  IRAMM  respondents  also  provided 
consequences  for  the  strategic  risk  scenarios  as  well  as  important  context  on  how  these  scenarios 
might  develop  in  the  future.  These  insights  provided  the  future  strategic  context  for  the  experts 
who  estimated  strategic  materials  shortfall  risks. 

Several  recent  IRAMM  enhancements  will  be  applied  in  the  next  cycle  of  RAMF-SM 
assessments  for  the  2015  Report  on  Stockpile  Requirements  to  Congress.  These  enhancements 
include  a  more  rigorous  accounting  of  the  future  risk-causing  scenario  space;  a  generalized 
approach  to  estimating  risk  for  a  subset  of  the  Challenge  Areas;  and  a  revised  set  of  Challenge 
Areas. 

The  IRAMM  approach  to  assessing  risk  is  a  rigorous  way  of  obtaining  DOD  senior 
leadership  views  on  strategic  national  security  strategy,  capabilities,  and  risk.  The  results  can  be 
used  in  a  multitude  of  ways  including  informing  debate  on  complex  issues,  identifying 
alternative  resource  strategies,  allocating  resources,  developing  rationale  for  decisions,  etc. 
Previous  applications  and  the  flexibility  of  IRAMM  suggest  more  areas  of  application  than  the 
current  use  in  RAMF-SM.  IRAMM  would  prove  useful  in  any  situation  where  decisions  are 
needed  to  mitigate  risk  across  disparate  areas  of  concern.  There  are  other  potential  uses  for 
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IRAMM  outside  the  arena  of  decision  making.  For  example,  it  could  provide  the  basis  for 
training  exercises  at  senior  military  schools. 
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1.  Introduction 


This  paper  describes  how  the  Integrated  Risk  Assessment  and  Management  Model 
(IRAMM)  is  used  in  a  decision-support  process  that  the  Department  of  Defense  (DOD) 
and,  more  broadly,  the  United  States  Government  (USG)  now  employ  to  assess  risks  and 
identify  cost-effective  risk  mitigation  strategies  in  the  important  area  of  non-fuel  strategic 
materials.  Many  of  these  materials,  such  as  beryllium,  high-perfonnance  fibers,  rhenium 
and  rare  earths,  are  essential  to  daily  life  in  the  United  States  and  many  defense  and 
critical  civilian  infrastructure  systems.  A  number  of  these  materials  may  be  in  short 
supply  during  national  emergencies.  This  paper  begins  with  an  overview  of  the  strategic 
material  decision-support  process  (the  Risk  Assessment  and  Mitigation  Framework  for 
Strategic  Materials  (RAMF-SM))  used  to  assess  risks  and  set  priorities  for  preparedness 
programs  designed  to  mitigate  potential  strategic  material  shortfalls.  Next,  the  paper 
describes  the  structure  of  IRAMM  followed  by  some  recent  IRAMM  results  used  in  the 
RAMF-SM  process  that  helped  produce  the  Strategic  and  Critical  Materials  2013  Report 
on  Stockpile  Requirements 1  to  Congress.  Several  recent  IRAMM  enhancements  that  will 
be  applied  in  the  next  cycle  of  RAMF-SM  assessments — for  the  2015  Report  on 
Stockpile  Requirements  to  Congress — arc  described  next.  The  paper  concludes  with 
observations  about  the  importance  of  IRAMM  within  RAMF-SM  as  well  as  the  potential 
utility  of  an  IRAMM-like  approach  to  support  U.S.  national  security  decisions. 

A.  Risk  Assessment  and  Mitigation  Framework  for  Strategic  Materials 
(RAMF-SM) 

The  RAMF-SM  process  helps  identify  cost-effective  strategies  for  mitigating 
potential  strategic  material  disruptions  and  shortfalls  using  a  risk  construct.  The  major 
steps  of  RAMF-SM  are  shown  in  Figure  1. 


U.S.  DOD,  Strategic  and  Critical  Materials  2013  Report  on  Stockpile  Requirements  (Washington,  DC: 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD  AT&L),  2013), 

(hereafter  cited  as  2013  Report  on  Stockpile  Requirements  or  2015  Report  on  Stockpile  Requirements, 
depending  on  the  year  of  report). 
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Figure  1.  RAMF-SM  Process  and  Steps 


RAMF-SM  has  six  major  steps.  Overall,  a  suite  of  interconnected  models,  hundreds 
of  data  bases  updated  for  each  new  RAMF-SM  cycle,  and  numerous  structured 
assessments  by  subject  matter  experts  (SME)  are  involved  in  its  application.  The  first  step 
in  RAMF-SM  is  to  select  the  set  of  materials  to  be  assessed  in  the  cycle.  The  second  step 
is  to  assess,  under  mandated  postulated  scenario  conditions,  the  extent  of  any  shortfalls  of 
supplies  of  materials  for  the  Nation  relative  to  critical  U.S.  military  and  civilian  demands 
for  them.  The  third  step  of  the  process  flow  diagram  shown  contains  two  separate  steps: 
the  assessment  of  shortfall  risks  for  unmitigated  materials  shortfalls,  and  the  IRAMM 
strategic  risk  assessment  (not  shown).  The  results  of  the  IRAMM  strategic  risk 
assessment  are  used  as  input  into  the  assessment  of  risks  for  unmitigated  materials 
shortfalls.  The  fourth  and  fifth  steps  involve  identifying  promising  options  (such  as 
developing  special  contingency  contracts  with  suppliers,  utilizing  substitute  materials, 
Federal  stockpiling)  to  mitigate  some  or  all  of  these  shortfalls/risks.  These  steps  include 
assessing  the  costs  and  effectiveness  of  each  mitigation  option,  and  of  combinations  of 
them,  in  terms  consistent  with  the  Office  of  Management  and  Budget  (OMB)  guidelines 
for  such  estimates.  The  sixth  and  final  step  involves  identifying  optimal  packages  of 
investments  in  mitigation  options — across  the  whole  set  of  shortfall  materials — under 
various  potential  budget  and  time  constraints.  The  primary  result  of  RAMF-SM  is  a  set  of 
recommended  mitigation  strategies,  including  investments. 

Much  of  the  RAMF-SM  structure  has  now  been  implemented  by  the  DOD  in 
preparing  its  recommendations  to  the  Congress  to  address  the  risk  to  the  Nation  of 
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potential  strategic  material  shortfalls.  The  structure  and  the  assessments  are  documented 

2 

in  the  2013  Report  on  Stockpile  Requirements  to  the  Congress. 

B.  Integrated  Risk  Assessment  and  Management  Model  (IRAMM) 

IRAMM  is  used  in  the  third  step  of  the  RAMF-SM  structure.  Step  3  involves  a 
material-specific  shortfall  assessment,  analyzing  the  risks  to  the  Nation  of  not  doing 
anything  to  mitigate  specific  material  shortfalls  in  the  case  of  interest.  A  full  description 
of  this  material  shortfall  risk  assessment  may  be  found  in  Appendix  12  of  the  2013 
Report  on  Stockpile  Requirements  to  the  Congress.  IRAMM  probabilities  for  the  Base 
Case4  scenarios  are  used  in  several  ways  in  RAMF-SM,  especially  in  estimating  the 
expected  costs  of  mitigation  options  that  would  be  incurred  if  the  Base  Case  should 
occur.  In  addition  to  providing  probabilities  for  the  Congressionally  mandated  Base  Case 
scenarios,  IRAMM  risk  findings  also  provide  the  future  strategic  context  for  the  experts 
who  estimate  strategic  materials  shortfall  risks. 

The  core  features  of  IRAMM  were  developed  and  documented  by  IDA  under 
separate  contract  with  the  DOD.  The  Institute  for  Defense  Analyses  (IDA)  Paper  P-44705 
provides  additional  background  on  IRAMM. 

IRAMM  has  five  key  features. 

1.  It  enables  coherent,  structured  elicitation  of  knowledgeable  evaluators’  estimates 
of  the  future  probabilities  of  various  significant  threat  scenarios  the  United 
States  may  confront  in  the  future,  some  of  which  may  feature  disruptions  of 
strategic  material  supplies  to  the  United  States. 

2.  It  helps  evaluators  identify  the  adverse  consequences  to  the  Nation  that  would, 
in  their  judgment,  likely  result  when  using  a  specified  set  of  forces,  military 
capabilities  and  related  preparedness  programs  to  address  a  threat,  or  set  of 
threats,  should  those  scenarios  actually  occur.  Consequences  for  each  potential 
scenario  are  explicitly  evaluated  within  IRAMM  along  three  categories 
(economic,  military,  and  political).  These  evaluations  are  then  integrated  by  the 
respondent,  facilitated  by  the  interviewer,  into  a  single,  ratio-scaled  metric  that 
enables  comparisons  among  scenarios  in  all  phases  of  the  IRAMM  elicitation. 

3.  It  allows  respondents  to  express  their  views  on  strategic  risk6  during  one-on-one, 
not-for  attribution  interviews.  Each  respondent  uses  a  common  risk  definition 


DOD,  2013  Report  on  Stockpile  Requirements,  Appendix  12. 

Ibid,  Appendix  12. 

Ibid.  Appendix  3. 

James  S.  Thomason,  IDA 's  Integrated  Risk  Assessment  and  Management  Model ,  IDA  Paper  P-4470 
(Alexandria,  VA:  IDA,  June  2009). 
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and  scales  for  estimating  consequences  so  that  the  results  can  be  used  in  a  group 
discussion  among  respondents  that  follows  the  interviews.  The  group  discussion 
is  facilitated  in  order  to  frame  areas  of  agreement  and  disagreement.  At  the  end 
of  the  group  discussion,  respondents  are  provided  an  opportunity  to  revise  their 
initial  risk  scores  in  light  of  the  evidence  and  insights  of  other  senior 
respondents  and  SMEs. 

4.  It  offers  a  means  by  which  many  promising  trades — puts  and  takes — may  be 
teed  up  across  the  entire  enterprise  in  terms  that  are  useful  to  senior  decision 
makers.  While  perhaps  not  perfect  in  this  regard,  IRAMM  terminology  and 
scales  may  offer  a  major  improvement  in  structured  decision  making. 

5.  It  provides  respondents  with  an  opportunity  to  identify  alternative  significant 
threats  to  the  Nation.  Respondents  may  assign  their  own  subjective  probability 
of  a  scenario’s  occurrence  and  their  individual  assessment  of  consequences 
should  it  occur.  IRAMM  is  not  just  a  mechanistic  application  of  the  standard 
formula  equating  risk  as  the  simple  product  of  probability  and  adverse 
consequences,  as  discussed  later  in  the  text. 

C.  Definition  of  Risk 

DOD  defines  risk  as  “probability  and  severity  of  loss  linked  to  hazards.”7  The 
IRAMM  methodology  is  concerned  with  strategic  risk  detennined  in  the  context  of 
various  force  structure  alternatives.  Accordingly,  strategic  risk  is  defined  to  be  the 
predicted  political,  economic,  and  military  losses  or  hazards  facing  the  United  States, 
based  on  the  expected  likelihood  and  character  of  future  events  and  conditions,  assuming 
currently  programmed  military  forces  (or  a  postulated  alternative  to  those  forces  and 
capabilities).  There  are  a  number  of  key  facets  to  this  definition  that  affect  how  strategic 
risk  (hereafter  referred  to  as  risk)  is  to  be  measured.  Note  that  risk  involves  future  events 
and  conditions.  Estimating  risk  is  not  simply  a  matter  of  analyzing  current  and  historical 
data;  rather,  it  involves  predicting  what  scenarios  might  happen  and  how  consequential 
they  will  be  if  they  do  happen.  Specifically,  it  requires  estimating  two  aspects  of  future 
events  and  conditions:  likelihood  and  character.  The  IRAMM  methodology  uses 

o 

subjective  probability  to  estimate  the  likelihood  of  future  events  and  conditions;  it  uses 


The  strategic  risk  in  IRAMM  is  not  the  same  as  the  material  shortfall  risk  estimated  in  the  2013  Report 
on  Stockpile  Requirements. 

Joint  Publication  1-02,  DOD  Dictionary  of  Military >  and  Associated  Terms  (Washington,  DC:  DOD, 
amended  15  December  2013),  230.  The  American  Heritage  Dictionary  defines  risk  similarly,  as  “the 
possibility  of  suffering  harm  or  loss.”  American  Heritage  Dictionary,  4th  Edition,  s.v.  “risk.” 

Subjective  probability  is  a  quantification  of  uncertainty  that  obeys  the  convexity  rule,  the  addition  rule, 
and  the  multiplication  rule.  See  George  Wright  and  Peter  Ayton,  Subjective  Probability  (Chichester: 
John  Wiley  and  Sons,  1994),  4-5. 
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economic,  military,  and  political  consequences  to  represent  the  character  of  future  events 
and  conditions. 
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2.  2012  Risk  Assessment  Exercise 


The  IRAMM  methodology  was  recently  used  in  the  preparation  of  a  report  from  the 
Secretary  of  Defense  to  the  U.S.  Congress  entitled  Strategic  and  Critical  Materials  2013 
Report  on  Stockpile  Requirements  9  The  Strategic  Materials  Office  of  the  Defense 
Logistics  Agency  (DLA)  sponsored  IDA  to  conduct  an  exercise  to  support  the  risk 
assessment  portion  of  the  analysis.  Specifically,  IRAMM  exercise  participants  estimated 
probabilities  of  occurrence  for  two  versions  of  the  Base  Case  set  of  scenarios  used  to 
assess  wartime  demand  and  supply  of  strategic  materials.  (See  sub-section  2.C.3)  In 
addition,  IRAMM  exercise  participants’  qualitative  descriptions  of  potential  strategic 
consequences  of  the  Base  Case  scenarios  (see  sub-section  2.C.2)  provided  strategic 
context  to  a  panel  of  materials  experts  who  contributed  separate  estimates  of 
consequences  of  specific  material  shortfalls.  Exercise  participants  also  estimated 
probabilities  of  occurrence  for  peacetime  supply  disruption  scenarios. 

A.  Background 

In  the  IRAMM  exercise,  participants  estimated  levels  of  risk  to  U.S.  national 
interests  in  four  distinct  categories  of  military  missions:  Major  Combat,  Irregular 
Warfare,  Homeland  Defense,  and  Global  Peacetime  Operations.  The  timeframe  for  the 
assessment  was  the  ensuing  decade,  from  2012-2021.  Participants  assumed  that  U.S. 
military  forces  in  the  future  would  be  configured  as  they  are  in  the  President’s  FY2013 
budget  and  associated  Future  Years  Defense  Program  (FYDP).  The  IRAMM  framework 
included  one-on-one,  not-for-attribution  interviews  with  senior  leaders  that  lasted 
approximately  one  and  a  half  hours  on  average. 

1.  Selection  of  Experts 

The  experts  chosen  for  this  exercise  all  had  significant  experience  in  the  field  of 
national  security.  Current/retired  military  members  of  the  group  had  achieved  the  rank  of 
flag  officer — the  majority  of  them  achieving  the  four-star  rank.  Current  and  fonner 
civilian  government  participants  were  chosen  based  on  their  experience  at  senior  levels  in 
the  Office  of  the  Secretary  of  Defense  and  similar  positions  in  other  governmental 


Much  of  this  chapter  is  based  on  the  Annex,  Strategic  Risk  Assessments,  to  Appendix  12  of  the  2013 
Report  on  Stockpile  Requirements . 
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agencies  responsible  for  national  security.  Members  with  an  asterisk  by  their  name  were 
currently  in  the  position  cited  at  the  time  of  the  interview. 

Experts  consulted  in  the  study  were:  Matthew  Beebe,  Deputy  Director,  Acquisition, 

Defense  Logistics  Agency;  Frank  Carlucci,  Secretary  of  Defense,  National  Security 

Advisor  to  the  President;  General  George  Casey,  Chief  of  Staff,  U.S.  Army;  General 

Kevin  Chilton,  Commander,  U.S.  Strategic  Command;  Dr.  David  Chu,  Undersecretary  of 

Defense  (Personnel  and  Readiness);  Admiral  Vernon  Clark,  Chief  of  Naval  Operations; 

Michael  Dominguez,  Acting  Secretary  of  the  Air  Force,  Principal  Deputy  Undersecretary 

of  Defense  (Personnel  and  Readiness);  Alan  Estevez,*  Assistant  Secretary  of  Defense 

(Logistics  and  Materiel  Readiness);  General  Carlton  Fulford,  Deputy  Commander,  U.S. 

European  Command;  General  Alfred  Gray,  Commandant,  U.S.  Marine  Corps;  General 

Michael  Hayden,  Director,  Central  Intelligence  Agency;  General  H.T.  Johnson,  Acting 

Secretary  of  the  Navy,  Commander,  U.S.  Transportation  Command;  Robert  Manning, 

•  •  •  .  ^  . 
Deputy  National  Intelligence  Officer,  Economic  Issues;  Deborah  McWhinney,  Chief 

Operating  Officer,  Citi  Global  Enterprise  Payments,  Citigroup;  Rear  Admiral  Richard 

•  •  •  .  .  sfc 

Porterfield,  Director  for  Intelligence,  U.S.  Pacific  Command;  Philip  Rogers,  Deputy 
Director,  Acquisition  Resources  Analysis,  Office  of  the  Secretary  of  Defense;  General 
Larry  Welch,  Chief  of  Staff,  U.S.  Air  Force;  Dr.  John  White,  Deputy  Secretary  of 
Defense. 

2.  Description  of  Challenge  Areas 

The  risk  assessments  were  structured  around  four  Challenge  Areas.  These  four 
Challenge  Areas  were  defined  so  that  together  they  covered  a  full  range  of  potential 
operations  conducted  by  the  U.S.  military.  The  Challenge  Areas  and  their  definitions 
used  in  this  exercise  are  shown  in  Figure  2. 
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Challenge  Areas 

|  Definitions 

Major 

Combat 

Operations  conducted  against  a  state  or  non-state  actor  that  possesses 
significant  military  capability.  This  area  should  account  for  risk  related  to  the 
use  of  WMD  during  the  course  of  major  combat. 
e.g.,  China,  North  Korea,  Iran,  Libya 

Irregular 

Warfare 

Stability  operations,  counterinsurgency,  peacekeeping,  or  counterterrorism 
operations  involving  significant  participation  ofU.S.  forces  in  combat  or 
prospective  combat. 

e.g.,  Iraq,  Afghanistan,  Bosnia,  Somalia 

Homeland 

Defense 

Protection  ofU.S.  sovereignty,  territory,  population,  and  critical  infrastructure 
against  external  threats.  This  area  should  delineate  among  risks  from  WMD, 
cyber  attack,  and  all  other  forms  of  external  attack  (except  those  directly 
related  to  Major  Combat). 

e.g.,  9/11,  missile  attack,  WMD  attack,  cyber  attack,  other  terrorist  attack 

Global 

Peacetime 

Operations 

Operations  conducted  to  influence  partners  and  adversaries.  This  area  should 
account  for  risks  related  to  changes  in  allied  or  adversary  military  capabilities, 
weapons  proliferation,  or  political  instability  that  are  contrary  to  U.S. 
peacetime  military  objectives  but  do  not  result  in  U.S.  combat  operations. 
e.g.,  presence,  deterrence,  building  partnership  capacity,  counter¬ 
proliferation,  freedom  of  navigation,  humanitarian  and  disaster  response. 

Figure  2.  Challenge  Area  Definitions 


Respondents  were  asked  to  identify  potential  future  scenarios  in  each  of  the  four 
Challenge  Areas.  For  each  scenario  identified,  they  were  asked  to  estimate  (1)  the 
likelihood  that  the  scenario  would  occur  in  the  next  ten  years;  and  (2)  the  consequences 
of  the  scenario  given  that  it  occurs  using  the  IRAMM  consequence  scale.  The  product  of 
these  two  parameters  generated  a  risk  score. 

3.  Consequence  Categories 

The  IRAMM  Consequence  Scale  Aid  (see  Figure  3)  is  used  by  respondents  to  help 
them  measure  consequences  consistently.  Consequences  are  measured  in  three  categories: 
economic,  military,  and  political.  The  consequence  scale  is  based  in  part  on  the  findings 
from  a  2000  study  co-chaired  by  General  Andrew  J.  Goodpaster.10  That  study  presented  a 
hierarchy  ofU.S.  strategic  interests  in  each  of  these  three  categories  with  the  highest  category 
defined  as  “vital”  (threatening  the  survival  of  the  United  States  as  a  sovereign  Nation),  and 
lesser  categories  defined  in  terms  of  decreasing  importance.  The  criteria  for  each  category 


The  Commission  on  America’s  National  Interests,  America ’s  National  Interests  (Cambridge,  MA: 
Belfer  Center  for  Science  and  International  Affairs,  Harvard  University,  July  2000). 
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identified  in  that  report  were  refined  to  the  bullet  form  that  appears  in  the  IRAMM 
consequence  scale  aid. 

The  consequence  scale  is  constructed  so  that  the  criteria  in  each  of  the  three  cells  in 
a  row  have  roughly  equal  consequences11  relative  to  their  category.  Since  there  were  five 
rows,  respondents  tended  to  assign  consequence  scores  in  five  equal-sized  bins,  from  0-20, 
20  40,  40-60,  60-80,  and  80-100.  However,  respondents  were  not  bound  to  the  values 
implied  by  the  scale  developers  and  were  free  to  assign  consequence  scores  in  accordance 
with  their  value  system  on  a  scale  of  0  to  100  or  more. 


Consequence  Scale  Aid 


Economic  Military  Political 


•4%  or  greater  cumulative  loss  in  GDP 

•  Extreme  semi-permanent  structural 
and  economic  costs, 

•  Capital  flows  massively  degraded  and/ 
or  dollar  collapses  jeopardizing  U.S. 
economic  foundation. 

•Alliances  and  economic  agreements 
terminated. 

•  Loss  of  more  than  10%  of  overall  military  force 

ceDabilitv  recoverv  longer  than  4  vears 

•  Covering  worldwide  mission  areas  adequately  is 
impossible 

•  Deterrence  severely  compromised  in  key  areas. 

•  Potential  international  condemnation  due  to  high  non- 
combatanl  casualties. 

•  Loss  of  confidence  in  military,  internally  and  externally 

•  The  U  S  seen  as  unreliable  by  multiple  allies  or 
coalition  partners  and  new  regional  secunty 
orders  emerge. 

•  Loss  of  credibility  as  guarantor  of  global 
security 

•  Allies  and  friends  create  their  own  nuclear 
arsenals  to  guarantee  their  secunty 

•  Competitors  become  increasingly  aggressive 
and  adversarial. 

Most 

Severe 

•  3%  cumulative  loss  m  GDP 

•  Severe  economic  costs  resulting  from 
trade  disruptions,  operational  factors, 
or  property  damage . 

•  Capital  flows  seriously  degraded  and' 
or  substantial  devaluation  of  dollar 

•Global  economy  stalled. 

•  Recovery  eventually 

•  Loss  of  5-10%  of  overall  military  force  capability; 
recovery  within  4  yrs 

•  Reduced  worldwide  mission  areas  commitment 

•  Deterrence  weak  in  key  areas. 

•  Critical  U.S.  vulnerability  revealed  to  all  from  military 
surprise. 

•  International  criticism  due  to  high  non-combatant 
casualties 

•  U.S.  strategic  influence  severely  degraded 

•  U.S  loses  credibility  in  one  or  more  key  regions 
of  the  world. 

•  One  or  more  competitors  takes  advantage  of 
perceived  U  S  weakness 

•  Some  coalitions  fail,  some  allies  turn  away  from 
the  U.S. 

•2%  cumulative  loss  m  GDP 

•  Serious  economic  costs  due  to  trade 
disruptions,  operational  factors,  or 
properly  damage. 

•  Capital  flows  degraded  and  .'or  value 
of  dollar  weakens 

•Economic  disruptions  possible,  but  no 
recession  follows. 

•  Reconstruction  of  key  economic 
capabilities  could  take  months 

•  Loss  of  1-5%  of  military  force  capability;  recovery 
within  18  months. 

•Worldwide  mission  areas  still  covered 

•  Overall  mission  success  not  questioned 

•  Deterrence  weaker,  but  still  strong. 

•  High  non-combatant  casualties. 

•  U.S.  weakened  as  major  global  political  broker. 

•  International  cooperation  with  U  S  put  at  nsk 

•  U.S.  credibility  weakened  with  one  or  more 
competitors 

•  U.S.  partners  doubt  U.S.  commitment  and  begin 
to  forge  separate  security  arrangements  or  seek 
unilateral  measures  to  guarantee  their  security, 

•  1%  cumulative  loss  m  GDP 

disruptions,  operational  factors,  or 
property  damage. 

•  Confidence  quickly  restored 
domestically  and  internationally 

•Worldwide  mission  areas  covered  adequately 
•  Low  or  predicted  non-combatant  casualties. 

•  Some  political  opposition  to  and  suspicion  of 

U  S  intentions  in  previously  friendly  countries 

•  Reduced  willingness  of  allies  and  friends  to 
cooperate  with  U.S.  on  other  international 
secunty  goals. 

•  Negligible  effect  on  GDP 

•  No  major  loss  of  military  force  capability  overall. 
•Worldwide  mission  areas  covered  adequately 

•  Low  or  predicted  non-combatant  casualties. 

•  Some  minor  political  opposition  to  and  suspicion 
of  U.S.  intentions  in  previously  fnendly 
countries 

Least 

Severe 

Figure  3.  Consequence  Scale  Aid 


IDA  subject  matter  experts  developed  the  criteria  used  in  the  scales.  Each  of  the  consequence 
categories  was  developed  as  an  ordinal  scale  with  the  severity  of  consequences  ordered  from  most  to 
least  in  each  category. 
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B.  Description  of  Elicitation  Protocol 

The  interview  begins  with  a  brief  overview  of  the  IRAMM  methodology.  The  heart 
of  the  elicitation  protocol  consists  of  requests  for  the  respondent  to 

•  identify  scenarios  within  each  Challenge  Area  that  cause  risk  in  the  time  period 
specified 

•  estimate  the  probability  of  occurrence  for  each  of  the  identified  scenarios 

•  estimate  the  consequences  associated  with  each  scenario  for  a  specified  force 
structure. 

Consequences  for  all  scenarios  identified  by  respondents  are  compared  to  the 
consequences  for  the  calibration  scenario,  and,  as  the  exercise  progresses,  to  previously 
estimated  scenario  consequences.  Respondents  are  asked  to  provide  a  ratio  value  for 
comparison  (e.g.,  a  given  scenario  is  “I/2”  as  consequential  as  the  calibration  scenario). 
The  summation  of  the  risk  scores  for  all  the  scenarios  in  a  Challenge  Area  is  a  measure  of 
the  overall  risk  for  that  Challenge  Area. 

1.  Consequence  Calibration 

Short  scenario  descriptions  were  defined  to  serve  as  calibration  points  for  the  top  and 
the  bottom  of  the  100-point  scale  (see  Figure  4  for  an  illustration). 
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Calibration  Points  for  Consequence  Estimates 

A  15  kiloton  nuclear  device  smuggled  in  a  skipping  container  detonates  in  the  New  York 
harbor  port,  killing  50.000.  Direct  and  indirect  damage  is  estimated  at  more  than  $600  billion 
(-4.0%  of  U.S.  GDP). 


In  the  absence  of  catastrophic  events.  U.S.  forces  maintain  a  steady-state  presence  and  level  of 
global  operations.  Civil  unrest  in  developing  countries  persists,  as  do  periodic  small-scale 
terrorist  attacks  aaainst  U.S.  allies  and  interests  overseas. 


Figure  4.  Consequence  Calibration  Points 


A  scenario  involving  a  nuclear  attack  on  the  U.S.  homeland  was  used  as  the 
calibration  scenario  and  assigned  an  overall  consequence  value  of  100.  This  does  not 
imply  that  this  scenario  scores  100  on  each  of  the  consequence  categories — political, 
military,  and  economic — should  it  occur.  Rather,  its  function  is  to  provide  a  starting  point 
for  respondents  as  they  assess  the  consequences  of  scenarios.  Another  scenario  depicting 
no  significant  military  events  over  the  2012-2021  decade  was  associated  with  the  bottom 
of  the  scale,  representing  a  consequence  score  of  effectively  0  on  the  100-point  scale. 
Initially,  respondents  were  asked  to  estimate  their  consequence  scores  for  the  scenarios 
they  identified  in  relation  to  these  calibration  points.  As  the  exercise  matures,  respondents 
can  use  previously  assessed  scenarios  as  points  of  pairwise  comparison  for  each  new 
scenario,  thus  helping  to  insure  the  internal  consistency  of  their  answers  as  well  as 
providing  the  fundamental  basis  for  using  the  product  of  probability  and  consequences  as 
a  risk  metric.  Respondents  deemed  almost  all  of  the  scenarios  they  identified  to  be  less 
consequential  than  the  high-end  of  the  calibration  scenarios.  In  practice,  nothing 


Using  pairwise  comparisons  to  induce  ratio-scaled  comparisons  based  on  ordinal  scales  is  described  in 
Ward  Edwards,  “How  to  use  Multi -attribute  Utility  Measurement  for  Social  Decision  Making,”  IEEE 
Transactions  on  Systems,  Man,  and  Cybernetics  SMC-7,  no.  5  (May  1977):  326-340. 
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prohibited  respondents  from  estimating  consequences  that  were  greater  than  the  100 
assigned  to  the  calibration  scenario,  and  a  few  respondents  chose  to  do  so. 

Respondents  used  the  scales  in  the  following  way.  For  each  scenario  they  identified 
within  a  given  Challenge  Area,  they  were  asked  to  provide  (1)  the  probability  that  the 
scenario  would  occur  in  the  next  decade;  and  (2)  the  political,  military,  and  economic 
consequences  of  the  scenario  should  it  occur  using  the  Consequence  Scale  Aid  and  the 
calibration  scenarios. 

The  risk  assessment  for  each  scenario  was  produced  in  accordance  with  the 
respondent’s  own  value  system.  That  is,  for  each  scenario  in  the  challenge  area, 
respondents  were  asked  to  estimate  the  political,  military,  and  economic  consequences 
separately;  the  respondents  were  then  asked  to  combine  them.  The  method  of 
combination  was  left  to  their  discretion,  although  most  chose  to  average  the 
consequences. 

Throughout  the  elicitation  process,  respondents  were  asked  to  provide  pairwise 
comparisons  between  consequence  estimates  for  new  scenarios  and  consequence 
estimates  for  previously  assessed  scenarios.  This  step  helps  ensure  consistency  in  their 
risk  judgments.  For  example,  a  respondent  who  had  estimated  an  overall  consequence 
score  of  40  would  be  asked  to  verify  that  this  scenario  was  twice  as  consequential  as  one 
they  had  assessed  earlier  to  have  a  score  of  20.  Similarly,  pairwise  comparisons  were 
made  for  risk  estimates  (the  product  of  probability  and  consequences)  of  different 
scenarios.  The  interviewer  would  also  ask  the  respondent  to  compare  total  risk  estimates 
for  a  just  completed  Challenge  Area  to  previously  completed  Challenge  Areas.  For 
example,  at  the  end  of  the  elicitation  for  the  Irregular  Warfare  (IW)  Challenge  Area,  a 
respondent  with  a  total  risk  score  of  30  for  IW  and  a  total  risk  score  of  10  for  Major 
Combat  Operations  (MCO)  would  be  asked  to  confirm  the  total  risk  for  IW  was  “three 
times  as  risky”  to  U.S.  vital  national  interests  as  the  MCO  Challenge  Area.  Thus,  at  the 
end  of  each  Challenge  Area,  the  respondent  was  given  the  opportunity  to  change  his  or 
her  risk  estimates  to  maintain  consistency  across  Challenge  Areas.  A  useful  tool  for 
helping  the  respondent  to  visualize  his/her  responses  was  the  risk  profile.  Figure  5  shows 
a  notional  risk  profile  for  Challenge  Areas  A  through  D. 
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Figure  5.  Notional  Risk  Profile 


2.  Order  of  the  Elicitation 

For  consistency,  the  Challenge  Areas  were  presented  in  the  same  order  for  each 
respondent,  as  follows:  Major  Combat  Operations,  Irregular  Warfare,  Homeland  Defense, 
and  Global  Peacetime  Operations.  This  order  was  intentional  and  important  to  the 
elicitation  protocol.  The  interviewer  starts  with  MCOs  as  all  the  scenarios  in  this 
Challenge  Area  are  discrete  events  with — as  it  turns  out — relatively  low  likelihood  of 
occurrence.  This  enables  the  use  of  event  trees,  a  structure  that  helps  the  respondent 
assess  probabilities  consistently.  Assessing  the  MCO  Challenge  Area  first  allowed  the 
respondents  to  become  familiar  with  the  idea  of  combining  the  two  elements  of  risk 
(likelihood  and  consequences)  before  they  were  introduced  to  the  phenomenon  of 
consequence  aversion.  In  contrast,  the  Global  Peacetime  Operations  Challenge  Area  was 
saved  for  the  end  of  the  exercise.  Scenarios  in  this  area  pose  a  much  different  challenge 
regarding  assessment  of  likelihood  as  they  often  involve  on-going  operations  that  lack 
well-defined  beginning  and  end  points.  Additionally,  these  operations  are  often  seen  as 
mitigating  risk  rather  than  adding  to  U.S.  risk.  The  IDA  team  found  that  the  prior  results 
of  assessing  risk  in  the  first  three  Challenge  Areas  helped  prepare  the  respondent  for 
assessments  of  these  complex  phenomena. 
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See  Chapter  3  of  this  paper  for  a  definition  of  event  trees. 
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C.  Results 


This  section  presents  some  major  results  from  the  2012  IRAMM  exercise.  The  first 
sub-section  presents  risk  profiles  that  provide  a  graphical  summary  of  respondents’ 
quantitative  views  of  risk  by  Challenge  Area.  The  second  sub-section  presents  a  summary 
of  qualitative  comments  that  support  respondents’  quantitative  assessments  of  risk.  The 
final  sub-section  discusses  how  the  probabilities  of  particular  scenarios  were  used  to  help 
produce  the  2013  Report  on  Stockpile  Requirements  to  Congress. 

1.  Risk  Profiles 

Figure  6  depicts  the  actual  risk  estimates  of  each  respondent  for  each  Challenge 

Area. 


100  — 

1.  Homeland  Defense  (36) 

80  2.  Irregular  Warfare  (22) 

3.  Major  Combat  (16) 

70  - 

4.  Global  Peacetime  Operations  (4) 

50  - - 


40 


Major  Combat  Irregular  Warfare  Homeland  Defense  Global  Peacetime 

Operations 


Figure  6.  Risk  Profile  Results 

This  graphic  was  useful  in  answering  the  question,  “How  did  I  compare  to  the  other 
respondents?”,  which  was  a  common  question  that  respondents  asked  following  the 
elicitation;  it  also  served  to  facilitate  discussions  among  the  respondents  in  the  panel 
session  that  followed  the  interview.  The  lines  connecting  the  risk  estimates  make  it  easier 
for  the  respondent  to  see  his/her  risk  profile  and  compare  it  to  those  of  other  respondents. 
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A  major  objective  of  IRAMM  risk  interviews  and  the  findings  they  produce  is  to 
structure  and  enrich  comprehensive  discussions  of  U.S.  strategic  risks  among  senior 
leaders.  This  objective  was  met  as  each  respondent  wrestled  with  the  question,  “Why  is 
my  risk  profile  different  from  the  risk  profile  of  other  respondents?”  The  line  on  the  chart 
labeled  Mean  Scores 14  offered  another  way  to  compare  themselves  to  their  peers.  The 
means  for  each  Challenge  Area  (the  number  in  parenthesis  after  each  Challenge  Area) 
also  provided  the  respondents  with  a  way  to  compare  the  magnitude  of  the  risk  estimates 
across  Challenge  Areas  using  a  ratio  scale. 

2.  Drivers  of  Risk 

Major  quantitative  results  can  be  summarized  in  the  following  four  observations 
referred  to  as  majority  viewpoints.  Each  majority  viewpoint  is  supported  by  qualitative 
comments  from  respondents  referred  to  as  drivers  of  risk. 

•  Majority  Viewpoint  #1:  Most  (67  percent)  participants  saw  more  strategic  risk 
in  Homeland  Defense  scenarios  than  any  other  category  of  military  missions. 

Drivers  of  Risk:  Participants  identified  more  risk  in  cyber-attack 
scenarios  than  in  other  types  of  scenario,  driven  by  judgments  that  a 
significant  successful  attack  is  both  relatively  likely  and  would  have 
relatively  large  consequences.  Nuclear  and  biological  attack  scenarios 
were  also  seen  as  relatively  high  risk,  with  severe  consequences  not  only 
in  terms  of  civilian  casualties,  but  also  in  enduring  psychological  and 
economic  effects. 

•  Majority  Viewpoint  #2:  Even  with  the  Iraq  war  over,  and  the  Afghanistan 
drawdown  underway,  most  (67  percent)  participants  saw  more  strategic  risk  in 
Irregular  Warfare  than  in  Major  Combat  Operations. 

Drivers  of  Risk:  Operations  in  Afghanistan  will  continue  to  be  costly,  and 
the  outcome  will  likely  be  unsatisfactory.  While  the  United  States  may 
choose  not  to  launch  another  stability  operation  of  the  same  scale  as 
Operation  Iraqi  Freedom  or  Operation  Enduring  Freedom,  irregular 
warfare  will  be  very  hard  to  avoid  altogether. 

•  Majority  Viewpoint  #3:  Most  (67  percent)  participants  saw  war  with  Iran  as  the 
riskiest  of  all  major  combat  scenarios. 

Drivers  of  Risk:  War  with  Iran  was  generally  considered  significantly 
more  likely  than  war  with  other  nation  states,  prompted  by  conflict  over 


Although  the  mean  is  generally  not  a  robust  estimator  of  the  central  tendency,  it  served  as  an 
appropriate  measure  in  this  instance.  Other  estimators  (the  median,  various  trimmed  means,  and 
Winsorized  means)  were  investigated;  they  did  not  vary  significantly  from  the  mean. 


16 


Iran’s  nuclear  program  or  Iranian  attempts  to  coerce  its  neighbors.  The 
global  retaliatory  reach  of  Iran’s  proxies  (e.g.,  Hezbollah)  also  elevated 
participants’  estimates  of  risk  related  to  war  with  Iran. 

•  Majority  Viewpoint  #4:  All  participants  believed  that  strong  global  military 
presence  and  engagement  is  important  to  U.S.  national  interests. 

Drivers  of  Risk:  If  budget  cuts  and  force  structure  reductions  reduced  the 
U.S.  military’s  ability  to  maintain  forward  presence  and  engagement,  then 
strategic  risk  would  increase  in  all  mission  areas  caused  by  eroding 
deterrence  against  regional  aggression,  reduced  confidence  of  allies  and 
partners  in  U.S.  commitment  to  their  security,  and  greater  difficulty 
maintaining  dialogue  and  cooperation  with  friends  and  potential  rivals 
alike. 

3.  Estimating  Probabilities  for  the  2013  Report  on  Stockpile  Requirements  to 

Congress 

The  2013  Report  on  Stockpile  Requirements  to  Congress  contains  the  DOD 
assessment  of  potential  U.S.  problems  regarding  strategic  and  critical  non- fuel  materials 
in  the  context  of  a  congressionally-mandated  Base  Case  planning  scenario.15  The  report 
recommends  material-specific  mitigation  strategies  for  the  problematic  materials.  To 
evaluate  the  utility  of  various  measures  for  mitigating  potential  shortfalls  of  materials  that 
could  occur  during  a  national  emergency,  DLA  Strategic  Materials  developed  a  risk 
assessment  and  shortfall  mitigation  cost-effectiveness  analytical  process.  The  first  step  in 
the  analysis  after  identifying  the  shortfall  risk  mitigation  measures  to  consider  is  to  assess  the 
existing  (unmitigated)  risk  arising  out  of  potential  material  shortfalls  during  the  NDS  Base 
Case  scenario.  Material  shortfall  risk  is  taken  to  be  the  product  of  the  probability  that  a 
material  shortfall  would  occur  and  the  consequence  to  the  nation  of  the  shortfall,  that  is, 

Shortfall  risk  —  P scenario  X  Cshortfall- 

In  the  course  of  the  2012  IRAMM  interviews  described  earlier,  respondents  provided 
estimates  of  the  probability  of  the  Base  Case  scenario. 16 


DOD,  2013  Report  on  Stockpile  Requirements,  1. 

Other  experts  (not  IRAMM  respondents)  estimated  the  consequences  of  material  shortfalls. 
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3.  Enhancements  to  the  IRAMM  Protocol 


Based  on  the  use  of  IRAMM  over  time  in  multiple  contexts,  IDA  has  identified 
three  areas  for  improvements  to  the  protocol  that  they  plan  to  implement  prior  to  the  next 
round  of  interviews  to  be  conducted  in  the  spring/summer  of  2014  in  preparation  for  the 
2015  Report  on  Stockpile  Requirements.  These  enhancements  are  described  in  the 
subsequent  sections. 

A.  More  Complete  Scenario  Space  for  Major  Combat  Operations 
(MCO) 

Though  the  respondents  were  experts  in  the  field  of  national  security  and  defense 
strategy  and  forces,  they  were  not,  in  general,  trained  in  the  mathematics  of  probability 
theory.  Most  people  understand  the  idea  of  probability  as  a  measure  of  one’s  belief  about 
an  uncertain  event;  however,  studies  have  shown  that  humans  find  it  difficult  to 
accurately  assess  probabilities.  The  IRAMM  methodology  employs  event  trees  to  the 
maximum  extent  possible  to  mitigate  known  difficulties  in  expert  probability  assessment. 
For  example,  if  one  were  to  ask  a  respondent  to  separately  assess  the  probability  of  four 
different  events  (A  through  D)  occurring  in  the  next  decade,  the  result  might  be  as 
follows:  P(A)  =  0.2;  P(B)  =  0.4;  P(C)  =  0.1;  P(D)  =  0.7.  Some  implications  of  this 
assessment  seem  relatively  clear:  for  example,  event  B  is  twice  as  likely  as  event  A;  event 
D  is  the  most  likely  of  the  four  events,  etc.  However,  there  are  implications  that  may  not 
be  obvious  to  some  respondents:  for  example,  there  is  a  non-zero  probability  that  more 
than  one  of  the  events  will  occur  in  the  next  decade. 

Event  trees  help  make  the  relationships  between  the  probabilities  of  different  events 
(scenarios)  and  combinations  of  those  events  clearer  to  the  respondent.  A  scenario 
description  contains  those  key  elements  of  a  conflict  that  a  respondent  needs  to  consider 
when  estimating  consequences  (e.g.,  United  States/allied  forces,  adversarial  forces,  goals 
and  constraints  for  both  sides,  etc.).  Since  respondents  identify  the  scenarios  that  he  or 
she  foresees  as  potentially  occurring  in  the  coming  decade,  the  information  that  describes 
those  scenarios  is  part  of  their  thought  process,  and  hence  can  differ  by  individual.  The 
elicitation  draws  out  and  records  their  scenario  information  and  thinking,  by  asking 


See,  for  example,  A.  Tversky  and  D.  Kahneman,,  “Judgments  under  Uncertainty:  Heuristics  and 
Biases,”  Science  185  (November  1961):  1124-1131.  See  also  Robin  M.  Hogarth,  Judgment  and 
Choice:  The  Psychology  of  Decisions  (Australia:  John  Wiley  and  Sons,  1980). 
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questions  such  as,  “how  do  you  see  the  scenario  playing  out”  and  by  comparing  their 
consequence  estimates  with  the  consequences  assigned  to  the  calibration  scenario  and  to 
other  scenarios  they  have  previously  assessed. 

To  build  an  event  tree,  the  respondents  were  first  asked  the  following  question: 
“What  is  the  probability  that  the  United  States  will  become  involved  in  one  or  more 
Major  Combat  Operations  during  the  next  ten  years?”  Respondents  typically  provided 
either  percentage  (e.g.,  40  percent)  or  chances  out  of  ten  (e.g.,  four  chances  out  of  ten). 
Respondents  were  then  asked  the  following  question  to  confirm  their  estimate:  “Does  this 
mean  that  you  believe  there  is  a  60  percent  chance  that  no  MCOs  will  occur  in  the  next 
ten  years?”,  thus  permitting  the  respondents  to  refine  their  answers.  The  initial  branches 
of  the  event  tree  are  called  Level  I  branches.  All  branches  that  stem  from  the  Level  I 
branches  are  called  Level  II  branches,  and  so  on  for  the  construction  of  the  complete  tree. 
All  of  the  probabilities  assigned  to  the  branches  for  Level  I  of  the  tree  must  sum  to  1.0. 
This  is  true  for  all  levels  of  the  tree.  Figure  7  is  an  example  of  an  event  tree  with  two 
branches  on  Level  I. 


Level  I 


Figure  7.  Level  I  of  an  Event  Tree 


All  probabilities  assigned  in  an  event  tree  are  conditional  based  on  the  state  of  the 
branch  from  which  they  stem.  The  tenninal  branches  of  an  event  tree  represent  the 
scenarios  that  the  respondent  has  identified,  and  the  probabilities  for  each  of  these 
scenarios  are  computed  by  multiplying  the  probabilities  along  the  path  of  branches  that 
lead  to  the  end  node  on  the  last  level.  By  construction,  all  the  scenarios  represented  on  the 
end  nodes  of  the  tree  are  mutually  exclusive.  Since  the  end  nodes  represent  specific 
scenarios,  respondents  may  find  it  easier  to  estimate  the  probability  for  the  scenario  as  the 
joint  probability  for  the  end  node  (the  product  of  the  probabilities  on  the  branch  path  to 
the  end  node).  This  is  illustrated  in  Figure  8. 
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Level  1 


Level  II 


Level  III 


Figure  8.  Event  Tree  for  MCOs 


During  the  interviews  conducted  in  2012,  respondents  were  asked  to  identify  all  of 
the  scenarios  in  the  coming  decade  that  they  could  foresee  as  causing  risk.  Although 
respondents  had  the  opportunity  to  specify  that  more  than  one  MCO  might  occur  in  the 
time  frame,  the  interviewer  did  not  specifically  elicit  the  probability  of  this  and 
consequently  respondents  did  not  do  this  frequently.  The  improved  protocol  (illustrated  in 
Figure  9)  is  designed  to  specifically  elicit  the  probability  of  multiple  MCOs. 


Level  I 


Level  II  Level  III 


Figure  9.  New  Fuller  Event  Tree  for  MCOs 


In  accordance  with  the  narrative  above,  respondents  will  be  formally  asked  the 
probabilities  for  the  branches:  I  only,  II  only,  etc.  Incorporating  these  extended  protocols 
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will  help  to  improve  the  rigor  of  accounting  for  all  scenarios,  but  will  also  add  time  to  the 
interview. 

B.  A  Different  Approach  for  the  Irregular  Warfare  (IW)  and 
Homeland  Defense  (HLD)  Challenge  Areas 

There  is  general  consensus  for  defining  risk  to  include  two  metrics:  likelihood  and 
character.  However,  there  is  no  consensus  on  how  to  quantitatively  measure  risk  in  a  way 
that  faithfully  includes  both  of  these  facets  of  risk.  Nevertheless,  it  is  common  to 
estimate  risk  by  using  the  product  of  probability  and  consequences.  This  is  how  risk  has 
been  quantified  in  the  IRAMM  methodology.  The  2012  interviews  highlighted  that  the 
product  is  not  the  best  approach  to  quantifying  risk  in  some  situations,  because  it  does  not 
appropriately  represent  the  respondent’s  underlying  beliefs  about  risk.  In  future  versions 
of  the  IRAMM  protocol,  the  product  of  probability  and  consequences  will  not  be  used 
exclusively  to  estimate  risk. 19 

Kaplan  and  Garrick  best  described  one  issue  with  using  the  product  of  probability 
and  consequences  when  they  stated,  “In  the  case  of  a  single  scenario  the  probability  times 
consequence  viewpoint  would  equate  a  low-probability  high-damage  scenario  with  a 
high-probability  low-damage  scenario — clearly  not  the  same  thing  at  all.”  There  are 
other  issues  such  as  sunk  cost  that  also  complicate  assessing  risk.  The  ensuing  paragraphs 
present  an  example  that  illustrates  these  and  other  issues,  by  comparing  the  risk 
associated  with  two  hypothetical  scenarios — one  in  the  Irregular  Warfare  Challenge  Area 
and  the  other  in  the  Homeland  Defense  Challenge  Area. 

In  the  IW  area,  some  scenarios  are  ongoing  and  have  been  for  years  (e.g.,  Operation 
Enduring  Freedom).  Hence  the  probability  of  the  scenario  occurring  is  1.0  and  one  can 
reason  that  the  consequences  of  remaining  operations  are  low — certainly  in  comparison 
to  the  consequences  that  have  already  been  realized  since  the  beginning  of  operations. 
Even  so,  respondents  are  reluctant  to  provide  extremely  low  consequence  scores  when 
the  lives  of  American  soldiers  are  still  at  risk  and  budgets  above  and  beyond  the  budget 
for  normal  peacetime  operations  are  still  being  allocated.  Another  influence  on 
respondents’  judgments  is  the  large  sums  of  money  and  lives  that  have  already  been 
spent.  In  these  cases,  respondents  typically  provided  a  consequence  score  of  5,  10,  or 
more  for  these  types  of  operations. 


See  for  example,  Bilal  M.  Ayyub,  Elicitation  of  Expert  Opinions  for  Uncertainty  and  Risks  (Boca 
Raton:  CRC  Press,  2001),  104.  See  also  Tim  Bedford  and  Roger  Cooke,  Probabilistic  Risk  Analysis: 
Foundations  and  Methods  (Cambridge:  Cambridge  University  Press,  2001),  10. 

In  the  2012  interviews,  respondents  were  free  at  the  end  of  their  interviews  to  adjust  their  calculated 
risk  scores  by  Challenge  Area,  and  some,  in  fact,  did  so. 

Stanley  Kaplan  and  John  B.  Garrick,  “On  the  Quantitative  Definition  of  Risk,”  Risk  Analysis  1,  no.  1 
(1981):  13. 
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In  the  HLD  arena,  consider  a  scenario  consisting  of  a  nuclear  device  detonated  in  a 
large  U.S.  metropolitan  area  to  which  the  respondent  assigns  a  probability  estimate  of 
1/100  and  a  consequence  score  of  100.  The  product  of  probability  and  consequences 
results  in  a  risk  score  of  1 .  When  compared  to  the  previous  Operation  Enduring  Freedom 
example  with  risk  scores  of  5  to  10,  many  respondents  felt  that  these  comparative  risk 
scores  did  not  accurately  represent  their  beliefs.  They  indeed  believe  that  the  risk 
associated  with  a  nuclear  detonation  in  a  large  U.S.  city  was  higher  than  the  risk 
associated  with  the  remaining  mission  in  Afghanistan. 

Kaplan  and  Garrick  go  on  to  assert  that,  “A  single  number  is  not  a  big  enough 
concept  to  communicate  the  idea  of  risk.”"  Accordingly,  they  propose  that,  “It  takes  a 
whole  family  of  curves  to  fully  communicate  the  idea  of  risk.”  These  findings  led  to  the 
development  of  the  following  chart  (Figure  10). 


H,  H 


c 


L,L 


The  two  X  s  represent  ordered  pairs  of  probability  (p)  and  consequences  (c) 
for  two  different  scenarios  resulting  in  roughly  equal  risk  scores  (p  x  c).  Yet,  due 
to  consequence  aversion,  many  respondents  view  the  scenario  associated  with 

/ 1  as  riskierthan  the  scenario  associated  with/ 2- 

Figure  10.  Consequence  Aversion 


While  it  is  true  that  information  is  lost  when  two  parameters  are  combined  into  one, 
it  is  a  necessary  sacrifice  in  the  context  of  the  IRAMM  interviews.  There  is  not  enough 


Ibid.,  14. 
Ibid. 
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time  to  develop  these  curves;  more  importantly,  this  technique  would  not  serve  the  main 
objective  of  the  interviews.  The  purpose  of  the  interviews  is  not  to  estimate  risk  per  se, 
but  rather  to  structure  conversations  among  senior  leaders  to  help  enable  better 
decisions.  Estimating  risk  as  a  single  number  helps  to  meet  this  objective.  Nevertheless, 
the  issue  of  equal  risk  scores  for  two  potentially  very  different  scenarios  needs  to  be 
addressed.  In  the  new  IRAMM  protocol,  the  chart  in  Figure  10  is  shown  to  respondents 
during  the  interview  along  with  a  statement  that  they  are  allowed  to  develop  risk  scores 
that  are  not  simply  the  product  of  probability  and  consequences.  They  initially  decide 
their  risk  score  using  the  product  formula;  they  are  free  to  adjust  this  score  based  on 
pairwise  comparisons  with  other  scenarios.  This  further  implies  that  the  sequence  of 
Challenge  Areas  used  in  the  interview  should  start  with  scenarios  that  lend  themselves  to 
assessing  risk  on  the  basis  of  the  product  of  probability  and  consequences  (e.g.,  MCOs). 
Risk  for  MCO  scenarios  can  then  become  the  basis  for  comparing  risks  in  the  context  of 
the  curves  in  Figure  10. 

C.  Changes  to  Challenge  Areas 

Two  changes  are  being  made  to  the  2012  Challenge  Areas.  One  is  dropping  the 
Challenge  Area  called  Global  Peacetime  Operations.  As  the  name  suggests,  this 
Challenge  Area  included  operations  that  served  to  strengthen  the  U.S.  defensive  posture 
through  activities  like  military-to-military  cooperation  obtained  from  activities  such  as 
Joint  exercises  and  training  with  foreign  militaries.  These  and  other  similar  activities  that 
fell  into  this  challenge  area  were  viewed  as  opportunities  rather  than  risks.  That  is,  the 
consequences  were  positive  and  could  not  be  measured  on  the  IRAMM  consequence 
scales.  Since  IRAMM  focuses  on  risk,  this  Challenge  Area  was  dropped. 

Another  change  is  to  divide  the  Homeland  Defense  Challenge  Area  into  two 
components:  Weapons  of  Mass  Destruction  (WMD)  against  the  Homeland,  and  Cyber 
Risks  to  the  Homeland.24  One  of  the  principal  drivers  for  this  change  is  the  recent 
emphasis  the  USG  is  placing  on  cyber  threats  to  the  U.S.  infrastructure.  This  change 
highlights  one  of  the  features  of  IRAMM;  Challenge  Areas  are  developed  to  support  the 
needs  of  the  senior  decision  maker. 


As  mentioned  in  Chapter  2,  the  interviews  also  produced  estimates  of  the  probability  of  the  Base  Case 
scenario  for  the  2013  Report  on  Stockpile  Requirements. 

There  are,  of  course,  other  threats  to  the  homeland  than  just  attacks  involving  WMD  and  cyber.  In  the 
2012  interviews,  threats  to  the  homeland  exclusive  of  these  categories  were  elicited;  however,  the  risk 
assigned  by  respondents  to  these  other  threats  was  insignificant  compared  to  WMD  and  cyber. 
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4.  Origin,  Value,  and  Other  Applications  of 

IRAMM 


In  early  2004,  three  offices  in  the  Office  of  the  Secretary  of  Defense  (OSD), 
Program  Analysis  and  Evaluation  (PA&E),  Acquisition,  Technology,  and  Logistics 
(AT&L),  and  Policy,  along  with  the  Joint  Staff,  J-8,  asked  IDA  to  develop  a  cross¬ 
capabilities  assessment  and  integrated  risk  management  analytic  framework  that  could  be 
used  to  compare  alternative  force  mixes.  The  result  was  IRAMM. 

The  premise  of  the  IRAMM  methodology  is  that  the  goal  of  national  security  and 
defense  strategy  is  to  mitigate  overall  risk  to  the  Nation,  and  therefore,  the  concept  of  risk 
to  the  Nation  is  the  appropriate  vehicle  for  a  process  that  supports  strategic  decision 
making.  Accordingly,  a  formalized,  quantified  measurement  of  strategic  risk  can  be  one 
useful  means  for  discriminating  among  alternative  policy  and  program  options. 
Establishing  a  viable,  comprehensive  process  for  senior  decision  makers  to  compare  the 
relative  merits  of  alternative  policy  and  program  options  in  mitigating  risks  to  the 
Nation — strategic  risk — as  they  are  making  major  defense  program  decisions  has  been 
the  chief  goal  of  IDA  researchers  in  building  IRAMM. 

The  approach  to  assessing  risk  described  in  this  paper  is  a  rigorous  way  of  obtaining 
DOD  senior  leadership  views  on  strategic  national  security  strategy,  capabilities,  and  risk. 
The  results  can  be  used  in  a  multitude  of  ways  including  informing  the  debate  on 
complex  issues,  identifying  alternative  resource  strategies,  allocating  resources, 
developing  rationale  for  decisions,  etc. 

Previous  applications  and  the  flexibility  of  IRAMM  suggest  more  areas  of 
application.  IRAMM  would  prove  useful  in  any  situation  where  decisions  are  needed 
across  disparate  areas  of  concern.  The  IRAMM  methodology  provides  a  hedge  against 
the  tendency  to  sub-optimize  in  each  area  of  concern  rather  than  consider  the  entire 
scenario  space.  IRAMM  accomplishes  this  by  providing  a  common  metric  that  can  be 
used  to  compare  options  associated  with  each  area  of  concern. 

In  addition  to  providing  a  common  framework  with  common  metrics  and  tenns  for 
the  advisors  to  a  decision  maker,  IRAMM  can  simultaneously  serve  as  a  team-building 
exercise.  The  interviews  and  the  subsequent  group  discussion  provide  a  forum  for  inter¬ 
agency  dialogue. 
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A  key  feature  of  the  elicitation  protocols  used  in  IRAMM  is  that  it  can  be  structured 
to  conform  to  and  support  the  needs  of  any  senior  decision  maker  in  a  hierarchical 
command  or  management  framework.  The  overarching  goals  of  the  IRAMM  elicitation 
framework  are  (1)  to  provide  a  common  value  system  within  which  the  principal  advisors 
in  the  management  framework  can  express  their  views  on  risk;  (2)  to  create  an 
environment  for  these  senior  advisors  to  express  and  share  their  viewpoints',  (3)  to  enable 
a  focused  discussion  among  these  advisors  that  provides  an  opportunity  to  express 
differences  of  opinion  with  the  potential  to  bidld  consensus  among  the  participants;  and 
(4)  to  synthesize  the  discussions — both  agreements  and  disagreements — to  provide  a 
basis  for  the  senior  decision  maker  to  issue  clear  guidance  on  risk  mitigation  priorities. 

There  are  other  potential  uses  for  IRAMM  outside  the  arena  of  decision  making.  For 
example,  it  could  provide  the  basis  for  a  training  exercise  at  senior  military  schools.  The 
exercise  would  help  educate  the  officers  in  the  value  of  structured  approaches  to  making 
decisions.  It  would  also  broaden  their  thinking  on  the  risks  to  the  Nation  as  well  as 
approaches  to  mitigate  those  risks. 

By  January  2015,  the  Secretary  of  Defense  is  once  again  required  by  law  (Section 
14b  of  the  Stock  Piling  Act)25  to  submit  an  assessment  to  the  Congress  of  strategic  and 
critical  material  risks  to  the  Nation  as  well  as  appropriate  strategies  and  recommendations 
to  mitigate  them,  including  but  not  limited  to  acquiring  Federal  inventories  for  the 
National  Defense  Stockpile.  To  prepare  these  assessments  and  recommendations,  DOD  is 
employing  the  RAMF-SM  framework,  and  the  IRAMM  strategic  risk  evaluations  are  an 
integral  part  of  RAMF-SM.  Toward  this  end,  a  new  round  of  one-on-one  IRAMM 
elicitations  (and  one  or  more  group  sessions),  will  be  conducted  in  2014  utilizing  the  new 
IRAMM  protocol  described  in  this  paper.  The  results  of  these  IRAMM  evaluations,  both 
by  senior  decision  makers  and  other  subject  matter  experts,  will  explicitly  inform  the 
RAMF-SM  assessments  and  the  2015  Report  on  Stockpile  Requirements  to  Congress. 


Section  14.,  Biennial  Report  on  Stockpile  Requirements,  [50  U.S.C.  98h— 5],  Strategic  and  Critical 
Materials  Stock  Piling  Act,  as  Amended  through  P.L.  1 12-239,  enacted  January  2,  2013. 
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AT&L 

Acquisition,  Technology,  and  Logistics 
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U.S.  Department  of  Defense 
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Future  Years  Defense  Program 
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Homeland  Defense 
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Institute  for  Defense  Analyses 

IRAMM 

Integrated  Risk  Assessment  and  Management  Model 

IW 

Irregular  Warfare 

MCO 

Major  Combat  Operation 

NDS 

National  Defense  Stockpile 

OMB 

Office  of  Management  and  Budget 

OSD 

Office  of  the  Secretary  of  Defense 

PA&E 

Program  Analysis  and  Evaluation 

RAMF-SM 

Risk  Assessment  and  Mitigation  Framework  for  Strategic 
Materials 

SME 

Subject  Matter  Expert 

U.S. 

United  States 

USG 

U.S.  Government 

WMD 

Weapon  of  Mass  Destruction 
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